Skip to content

test: add integration tests with in-memory Phoenix - #36

Merged
zurfjereluhmie merged 1 commit into
mainfrom
test/integration-tests
Apr 22, 2026
Merged

test: add integration tests with in-memory Phoenix#36
zurfjereluhmie merged 1 commit into
mainfrom
test/integration-tests

Conversation

@zurfjereluhmie

@zurfjereluhmie zurfjereluhmie commented Apr 22, 2026

Copy link
Copy Markdown
Owner

Tests

Uploader (5 tests)

  • Creates one dataset per unique tag
  • Datasets have correct example counts
  • Skip mode is idempotent (no duplicates)
  • Append mode adds examples to existing datasets
  • Custom input/output keys are preserved in examples

Runner (7 tests)

  • Dry run completes without error
  • Full run returns experiment results
  • Evaluator execution during run
  • Name filter selects specific experiments
  • Missing dataset exits with code 1
  • Concurrent execution (concurrency=2)
  • Custom prefix and metadata

Architecture

  • Session-scoped Phoenix fixture: one server per test session on a random free port
  • Unique dataset names (uuid suffix) to avoid conflicts between tests sharing the same server
  • @pytest.mark.integration marker: tests are excluded by default (addopts = "-m 'not integration'" in pyproject.toml)
  • CI job: dedicated test-integration job added to ci.yml, included in the ci-success gate
  • Makefile: make test-integration target

Running

make test-integration         # run integration tests only
make test                     # unit tests only (default, skips integration)
uv run pytest tests/ -m integration   # manual

- Add tests/integration/ with 12 tests covering the full upload and
  experiment lifecycle against a real in-memory Phoenix instance
- Uploader tests: create per tag, example counts, skip idempotency,
  append adds examples, custom input/output keys
- Runner tests: dry run, full run, evaluator execution, name filter,
  missing dataset exit, concurrent execution, custom prefix/metadata
- Session-scoped phoenix_server fixture (launch_app on a free port)
- Tests use unique dataset names (uuid suffix) to avoid conflicts
- Gate tests behind @pytest.mark.integration marker
- Default pytest config excludes integration tests (-m 'not integration')
- Add test-integration CI job and Makefile target
- Add test-integration to ci-success gate
@zurfjereluhmie
zurfjereluhmie force-pushed the test/integration-tests branch from 981cba9 to 8c22e99 Compare April 22, 2026 12:58
@zurfjereluhmie
zurfjereluhmie merged commit a49c8bb into main Apr 22, 2026
10 checks passed
@zurfjereluhmie
zurfjereluhmie deleted the test/integration-tests branch April 22, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant